6 links tagged with all of: decision models + classification
Click any tag below to further narrow down your results
Links
Baseten's research lab released two open-source decision models, mammoth-1 (27B) and elephant-1 (9B), built to classify and route inputs by returning probability scores instead of generating text. Both are free to try for a week on Baseten's Model APIs.
- mammoth-1 ranks #1 of 208 decision models on the Jev Decision Index, and elephant-1 is the top 9B model there.
- Decision models return a probability over possible answers rather than writing text, which makes them faster and cheaper than generative LLMs for tasks like ticket routing and spam detection.
- Confidence thresholds let you auto-handle high-certainty cases and escalate uncertain ones to a human.
- Both were fine-tuned from Qwen3.5 on 6.21 million decision-making examples, including training to say "I don't know."
The author compares newer decision models against LLM judges for automated evals on a leasing-agent failure task. The main takeaway is that the default 0.50 threshold often leaves performance on the table, and that the scores need tuning and calibration before you trust them.
- The default 0.50 decision threshold can miss nearly all failures; tuning it per model, using F1 with an 80% precision/recall floor, improves results.
- OpenAI Decisions had the best separation (F1 about 0.87) but cost about 4.5 times more than Perplexity Decider (F1 about 0.79) per 100 eval runs.
- Raw model probabilities are often poorly calibrated; Platt scaling helped Perplexity, Liquid d1, and Clef Flash, but made OpenAI's slightly worse.
Musubi released PolicyLM-1.7B, a lightweight decision model that applies content policies written in plain English to messages in under 50 milliseconds without needing retraining when policies change. The model costs and runs as fast as traditional AI classifiers but offers the flexibility of modern language models to handle complex moderation rules.
- PolicyLM-1.7B can process content policies in plain English and make binary moderation decisions in under 50 milliseconds, matching the speed and cost of existing classifier systems while avoiding the need for retraining when policies shift.
- Decision models output predetermined outcome probabilities rather than text, making them faster and cheaper than large language models while keeping transformer architecture flexibility — a trend that started with TypeSafe AI's Jev in September, followed by OpenAI and Amazon releases.
- The model gives platform teams a scalable way to label and understand content at scale without building custom training pipelines for each policy change, addressing the challenge of moderating exponentially growing content volumes.
Cloudflare open-sourced two decision models called Clef and Clef-flash that classify inputs into typed, structured outputs with probabilities — useful for routing customer support tickets, identifying phishing domains, or triggering automated agent actions. The models outperform Typesafe's Jev system on speed and benchmarks while supporting images and a 64k context window.
- Clef processes decisions 2x faster than general LLMs (2.2s vs 4.7s in Cloudflare's threat intelligence workflow) because it scores schema choices in parallel rather than generating text token-by-token.
- The models beat Jev on most Jev Decision Index benchmarks and Typesafe's own eval suite, with Clef-flash offering exceptional speed-to-quality tradeoffs; median latency is 209ms for Clef and 38.8ms for Clef-flash.
- Both models are open-sourced on Hugging Face under Apache 2.0, hosted on Workers AI at Cloudflare's edge for low network latency, and fully API-compatible with Jev for easy switching.
Strands released Strands Decider 2B, a 2-billion-parameter model designed to quickly pick between predefined options rather than generate arbitrary text. It runs locally in ~115ms and ranks third among 2B-class decision models on accuracy and calibration benchmarks.
- Decision models sacrifice flexibility for speed and reliability: they choose from given options and assign confidence scores, making them 100x faster than LLMs for simple classification tasks but unsuitable for text generation, coding, or reasoning.
- The 2B parameter size hits a sweet spot for local experimentation—small enough to run on consumer hardware, large enough to handle meaningful work like tool selection, routing, and guardrails in agentic workflows.
- Strands open-sourced the full model, training data, and 19 iterations of architecture refinements on GitHub and Hugging Face, explicitly inviting community contributions to the decision model space.
TypeSafe AI released Jev, a model that takes text input but outputs confidence scores and probability distributions instead of generated text. It's designed for classification tasks like spam detection and ranking, costs $0.042 per million input tokens with free output, and has already spawned open-source recreations and creative experiments.
- Jev returns structured probabilistic outputs (yes/no scores, rankings, category choices) rather than text, making it fast and cheap—you only pay for input tokens, not output.
- The model trades interpretability for simplicity: you get a floating-point score but no explanation of which signals drove the decision, raising serious concerns about hidden bias in high-stakes use cases like hiring.
- The community has already built unconventional projects with it (a chat model, a game player, a left-pad implementation) and created open-weight alternatives like Kev within days of release.